Dynamics of Neural Networks with Continuous Attractors 
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We investigate the dynamics of continuous attractor neural networks (CANNs). Due to the 
translational invariance of their neuronal interactions, CANNs can hold a continuous family of 
stationary states. We systematically explore how their neutral stability facilitates the tracking 
performance of a CANN, which is believed to have wide applications in brain functions. We develop 
a perturbative approach that utilizes the dominating movement of the network stationary states in 
the state space. We obtain results on the maximum speed for a moving stimulus to be trackable, 
and the reaction time to catch up an abrupt change in stimulus. 
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Understanding how the dynamics of a neural network 
is shaped by the network structure, and consequently 
facilitates the functions implemented by the neural sys- 
tem, is at the core of using physical models to elucidate 
brain functions Traditional models of attractor neu- 
ral networks (ANNs), such as those based on the Hop- 
field model are powerful for elucidating the compu- 
tation and the memory retrieving processes in the brain. 
When external noisy inputs are presented, the dynamics 
of ANNs will reach an attractor that is highly correlated 
with the memories stored in the network. Each attractor 
has its own basin of attraction well separated from the 
others. These models successfully describe the behavior 
of associative memories, and stimulated an upsurge of 
studies on the retrieval behavior of the models Q . How- 
ever, other than memory retrieval, many other computa- 
tional issues of the brain have not been analyzed to the 
same extent. 

Recently, a new type of attractor networks, called con- 
tinuous attractor neural networks (CANNs) , has received 
considerable attention [1 [E S S i, i, El, III [11 III, 
[isj . These networks possess a translational invariance of 
the neuronal interactions. As a result, they can hold a 
family of stationary states which can be translated into 
each other without the need to overcome any barriers. 
Thus, in the continuum limit, they form a continuous 
manifold in which the system is neutrally stable, and 
the network state can translate easily when the external 
stimulus changes continuously. Beyond pure memory re- 
trieval, this endows the neural system with a tracking 
capability. 

However, existing literature mainly focused on using 
CANNs to illustrate pattern formation, movement con- 
trol, orientation tuning, spatial navigation, head direc- 
tion, population decoding and working memory [^, 
[Til , [iq . Despite these initial successes, the conclusions 
are often based on simulation results, and more rigorous 
analyses of the tracking behavior are still lacking. The 
relation between the tracking behavior and the energy 
landscape was not well understood. 

In this Letter we propose, as far as we know, the first 
analytical solution of a model of CANNs to its tracking 



dynamics, that permits systematic perturbative improve- 
ment. We display clearly how the dynamics of a CANN 
is decomposed into different modes, and which of them 
dominates the tracking behaviors of the network. The 
solution is reminiscent of those of solitons in nonlinear 
dynamics, and by using the eigenfunctions of the quan- 
tum harmonic oscillator as the basis, a time-dependent 
perturbation analysis can be applied systematically. We 
expect our method provides a general tool for the the- 
oretical study of CANNs. Our work also generates new 
predictions on the tracking behaviors of CANNs, namely, 
the maximum tracking speed to moving stimuli, and the 
reaction time to sudden changes in external stimuli, both 
are testable by experiments. 

Specifically, we consider a one-dimensional continuous 
stimulus being encoded by an ensemble of neurons. For 
example, the stimulus may represent the moving direc- 
tion, the orientation, or a general continuous feature of an 
external object. Let U{x, t) be the synaptic input at time 
t to the neurons with preferred stimulus of real-valued 
X. We will consider stimuli and responses with correla- 
tion length a much less than the range of x, so that the 
range can be effectively taken to be (— oo, oo). The firing 
rate r{x, t) of these neurons increases with the synaptic 
input, but saturates in the presence of a global activity- 
dependent inhibition. A solvable model that captures 
these features is given by 



r{x,t) = 



U{x,tf 



1 + kpJ dx'U{x',ty 



(1) 



where p is the neural density, and fc is a small posi- 
tive constant controlling the strength of global inhibition. 
The dynamics of the synaptic input U{x, t) is determined 
by the external input IeyA(x,t), the network input from 
other neurons, and its own relaxation. It is given by 



dU{x,t) 
Jt 



Icxt{x,t) + p I dx'J{x,x')r{x',t) — U{x,t), 

(2) 

where t is the time constant, which is typically of the 
order 1 ms, and J(x,x') is the neural interaction from 
x' to x. The key characteristic of CANNs is the transla- 
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FIG. 1: The canyon formed by the stationary states projected 
onto the subspace formed by feijo and f)o|o- Motion along the 
canyon corresponds to the displacement of the bump (inset). 



tional invariance of their neural interactions. In our solv- 
able model, we choose Gaussian interactions with a range 
a, namely, J{x,x') — exp[— (x — x')^/(2o^)] J/\/27ra2. 
CANN models with other neural interactions and inhi- 
bition mechanisms have been study [1, H, d, . How- 
ever, our model has the advantage of permitting a sys- 
tematic perturbative improvement. Nevertheless, the fi- 
nal conclusions of our model are qualitatively applicable 
to general cases. 

We first consider the intrinsic dynamics of the CANN 
model in the absence of external stimuli. For < A; < 
kc = pJ^/(8\/27ra), the network holds a continuous fam- 
ily of stationary states, which are 



U {x\z) = C/q exp 



{x- z) 



21 



(3) 



where C/q = [l + {l-k/kcY/'^]J / {A^ak). These station- 
ary states are translationally invariant among themselves 
and have the Gaussian bumped shape peaked at arbitrary 
positions z. 

The stability of the Gaussian bumps can be studied 
by considering the dynamics of fluctuations. Consider 
the network state U{x,t) — U{x\z) +SU{x,t). Then we 
obtain 

T-^SU(x,t)= [ dx'F{x,x')SU(x',t) - SU(x,t), (4) 
at J 

where the interaction kernel is given by F(x, x') = 
p J dx"J{x,x")dr{x")/dU{x'). To compute the eigen- 
functions and eigenvalues of the kernel F{x,x'), we 
choose the wave functions of the quantum har- 
monic oscillators as the basis, namely, Vn{x\z) — 
exp(-CV2)i?„(0/v/(27r)i/2an!2», where ^ = (a; - 
a)/(\/2a) and Hn{£,) is the n**^ order Hermite polyno- 
mial function. Indeed, the ground state of the quantum 
harmonic oscillator corresponds to the Gaussian bump, 
and the first, second, and third excited states correspond 
to fluctuations in the peak position, width, and skewness 
of the bump respectively. The eigenvalues of the matrix 
Fmn are given by Ao = l-(l-fc/fcc)^/^, and A„ = 1/2""^ 
for n > 1. Since Ai = 1 and all other eigenvalues are less 
than 1, the stationary state is neutrally stable in one 
component, and stable in all other components. 



The eigenfunctions of F correspond to the various dis- 
tortion modes of the bump, the first two of which are 
particularly important. (1) The eigenfunction for the 
eigenvalue Aq is vq{x\z), and represents a distortion of 
the amplitude of the bump. As we shall see, amplitude 
changes of the bump affect its tracking performance. (2) 
Central to the tracking capability of CANNs, the eigen- 
function for the eigenvalue 1 is vi{x\z) and is neutrally 
stable. We note that vi{x\z) oc dvo{x\z)/dz, correspond- 
ing to the shift of the bump position among the station- 
ary states. This neutral stability is the consequence of 
the translational invariance of the network. It implies 
that when there are external inputs, however small, the 
bump will move continuously. This is a unique prop- 
erty associated with the special structure of a CANN, 
not shared by other attractor models. Other eigenfunc- 
tions correspond to distortions of the shape of the bump. 
For example, the eigenfunction for the eigenvalue 1/4 is 
■y/6/ 7v3 {x\z) + y/l/7vi{x\z), and corresponds to askewed 
distortion of the bump. 

It is instructive to consider the energy landscape in 
the state space of a CANN. Since Fmn is not symmet- 
ric, a Lyapunov function cannot be derived for Eq. ([4]). 
Nevertheless, for each peak position z, one can define an 
effective energy function E\z — J2ni'^ ~ '^'O^nlz/^i where 
hn\z is the overlap between lJ{x) — and the n*'' 

eigenfunction of F centered at z. Then the dynamics in 
Eq. ([4]) can be locally described by the gradient descent of 
E\z\Ti the space of 6„|2. Since the set of points hn\z = 
for n ^ 1 traces out a line with = in the state space 
when z varies, one can envisage a canyon surrounding the 
line and facilitating the local gradient descent dynamics, 
as shown in Fig. 1. A small force along the tangent of 
the canyon can move the network state easily. This illus- 
trates how the landscape of the state space of a CANN 
is shaped by the network structure, leading to the neu- 
tral stability of the system, and how this neutral stability 
shapes the network dynamics. 

Next, we consider the network dynamics in the pres- 
ence of a weak external stimulus. Suppose the neural 
response at time t is peaked at z{£). Since the dynam- 
ics is primarily dominated by the translational motion 
of the bump, with secondary distortions in shape, we 
may develop a time-dependent perturbation analysis us- 
ing {vn{x\z{ty)\ as the basis, and consider perturbations 
in increasing orders of n. This is done by considering 
solutions of the form 

oo 

C/(x, t) = lJ(x\z(t)) + an(t)vn(x\z{t)). (5) 

Furthermore, since the Gaussian bump is the steady- 
state solution of the dynamical equation in the absence of 
external stimuli, the neuronal interaction term in Eq. Q 
can be linearized for weak stimuli. Making use of the 
orthonormality and completeness of {w„(a;|z(t))}, we ob- 
tain from Eq. expressions for dan/dt for each order n 
of perturbation, each being driven by In{t), the overlap 
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between Iext{x,t) and the n'^ eigenfunction. Determin- 
ing z{t) by the center of mass of U{x,t), we obtain the 
self-consistent condition 



dz^2a ( /i +E:L3.oddv/"!!/("-l)!!(J»-««) \ 

(6) 

In practice, low order perturbations already yield very ac- 
curate results. Below, we consider the network dynamics 
for several kinds of external stimuli. 

1 ) Tracking a stationary stimulus: Consider the ex- 
ternal stimulus consisting of a Gaussian bump, namely, 
IcKt{x, t) = alio exp[— (x — 0o)^/4a^]. Perturbation up to 
the order n — 1 yields ai{t) = 0, [d/dt -I- (1 — Ao)/r]ao — 
aexp[— (zo — z)^/8a^]/r, and 



dz a 

- = -(zo-z)exp 



Rity 



(7) 



where R{t) = l + a f_^{dt' /t) exp[-{l - \o){t - t')/T - 



{zo - z{t')f 



Hence, the dynamics is driven by a 



pull of the bump position towards the stimulus posi- 
tion zq- When compared with the limiting expression 
for weak stimuli, which was considered in the dynamics 
of the moving bump in [Tsj . the pull is reduced by the 
factor R{t). This is due to the increase in amplitude of 
the bump, which slows down its response. 

Suppose, in addition to the bumped signal, there is 
also the Ii component consisting of a white noise of tem- 
perature T„. Then the noises shift the bump position 
randomly. In the absence of the bumped signal in the ex- 
ternal stimulus (a = 0), the peak position of the bumped 
response undergoes a random walk with (z(i)^) = 2Dt 
in the low noise hmit, where D = 4ar„/[(27r)^/^C/QT^] is 
the diffusion constant. In the presence of a weak bumped 
signal (a > 0), the peak of the bumped response expe- 
riences a pull which can be written as the gradient of a 
potential V{z) = — 4a^aexp[— (z — zo)^/8a^]/T. Thus, 
the probability distribution of z can be approximated by 
a Boltzmann distribution P{z) oc exp[— l/(z)/Z?] in the 
low noise limit. 

2) Tracking a moving stimulus: The tracking perfor- 
mance of a CANN is a key property that is believed to 
have wide applications in neural systems. Suppose the 
stimulus is moving at a constant velocity v, and the noise 
is negligible. The dynamical equation becomes identical 
to Eq. ([7]), with zq replaced by vt. Denoting the lag of 
the bump behind the stimulus by s = zq — z we have, 
after the transients, 



ds 
'dt 



v-g{s); g{s) = V- 



1 - Ao 



(8) 

The value of s is determined by two competing factors: 
the first term represents the movement of the stimu- 
lus, which tends to enlarge the separation, and the sec- 
ond term represents the collective effects of the neuronal 



recurrent interactions, which tends to reduce the lag. 
Tracking is maintained when these two factors match 
each other, i.e., v — g{s); otherwise, s diverges. 

Fig. 2(a) shows that the function g{s) is concave, and 
has the maximum value of ^max — 2aa/ (r^/e) at s = 2a. 
This means that if w > gmaxj the network is unable to 
track the stimulus. Thus, (?max defines the maximum 
trackable speed of a moving stimulus. Notably, gmax in- 
creases with the strength of the external signal and the 
range of neuronal recurrent interactions. This is reason- 
able since it is the neuronal interactions that induce the 
movement of the bump, jmax decreases with the time 
constant of the network, as this reflects the responsive- 
ness of the network to external inputs. 

On the other hand, for v < gmax, there is a stable and 
unstable fixed point, respectively denoted by si and S2 in 
Fig. 2(a). When the initial distance is less than S2, it will 
converge to si. Otherwise, the tracking of the stimulus 
will be lost. Figs. 2(b) and (c) show that the analytical 
results of Eq. ^ well agree with the simulation results. 

3) Tracking an abrupt change of the stimulus: Suppose 
the network has reached a steady state with an external 
stimulus stationary at t < 0, and the stimulus position 
jumps from to zq suddenly at t = 0. This is a typical 
scenario in experiments studying mental rotation behav- 
iors. We first consider the case that the jump size zq is 
small compared with the range a of neuronal interactions. 
In the limit of weak stimulus, the dynamics is described 
by Eq. ([7]) with R{t) ~ 1. We are interested in estimating 
the reaction time T, which is the time taken by the bump 
to move to a small distance 6 from the stimulus. In the 
limit of low noise, the reaction time increases logarithmi- 
cally with the jump size, namely, T « (r/a) ln(|zo|/0). 

When the strength a of the external stimulus is larger, 
improvement using a perturbation analysis up to n = 1 
is required when the jump size zq is large. This amounts 
to taking into account the change of the bump height 
during its movement from the old to new position. The 
result is identical to Eq. ([7]), with R{t) replaced by 



R{t) = 1 
X exp 



1 - Ao 
(1-Ao) 



exp 



(1-Ao)^ 



dt' 



[t-t') 



{zo-z{t')f 



(9) 



Indeed, R{t) represents the change in height during the 
movement of the bump. Contributions from the second 
and third terms show that it is highest at the initial and 
final positions respectively, and lowest at some point in 
between, agreeing with simulation results shown in Fig. 3 
inset. Fig. 3 shows that the first order perturbation over- 
comes the insufficiency of the logarithmic estimate, and 
has an excellent agreement with simulation results for 
Zo up to the order of 2a. We also compute the reaction 
time up to the fifth order perturbation, and the agree- 
ment with simulations remains excellent even when zo 
goes beyond 2a. This implies that beyond the range of 
neuronal interaction, tracking is influenced by the distor- 
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FIG. 2: (a) The function g(s) and the stable and unstable fixed points of Eq. (|8}. (b) The time dependence of the separation 
s starting from different initial values. Symbols: simulations with A'' = 200. Lines: fifth order perturbation. Dashed lines: si 
(bottom) and S2 (top), (c) The dependence of the terminal separation s on the stimulus speed v. Symbols: simulations with 
— 200. Solid line: prediction of 15]. Dashed line: first order perturbation. Parameters: a — 0.05, a = 0.5, t — 1, k — 0.5, 
p = N/{2-k), J = V2^. 
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FIG. 3: The dependence of the reaction time T on the new 
stimulus position zo- Symbols: simulations. Solid line: pre- 
diction of [isl. Dotted line: first order perturbation. Dashed 
line: fifth order perturbation. Parameters: as in Fig. 2. In- 
set: profiles of the bump between the old and new positions 
at = 7i"/2 in the simulation. 

tion of the width and skewed shape of the bump. 

To conclude, we have systematically investigated how 
the neutral stability of a CANN facilitates the tracking 
performance of the network, a capability which is be- 
lieved to have wide applications in brain functions. Two 
interesting behaviors are observed, namely, the maximum 
trackable speed for a moving stimulus and the reaction 
time for catching up an abrupt change of a stimulus. 



logarithmic for small changes and increasing rapidly be- 
yond the neuronal range. These two properties are as- 
sociated with the unique dynamics of a CANN. They 
are testable in practice and can serve as general clues 
in for checking the existence of a CANN in neural sys- 
tems. In order to solve the dynamics which is otherwise 
extremely complicated for a large recurrent network, we 
have developed a perturbative analysis to simplify the 
dynamics of a CANN. Geometrically, it is equivalent to 
projecting the network state on its dominant directions 
of the state space. This method works efficiently and 
may be widely used in the study of CANNs. Further- 
more, the special structure of a CANN may have other 
applications in brain functions, for instance, the highly 
structured state space of a CANN may provide a neural 
basis for encoding the topological relationship of objects 
in a feature space, as suggested by recent psychophys- 
ical experiments fv^., . We expect the mathematical 
framework developed in this study will be very valuable 
to explore these issues. 
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